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Description 
REAL TIME DATA STREAM PROCESSOR 

Background of Invention 
Field of the Invention 

[0001] The invention relates generally to real-time image pro- 
cessing systems. More particularly, methods and appara- 
tus for efficiently processing multi-format video streams 
including those derived from bi-directional, packetized, 
non-point to point data communications networks into a 
single format video stream suitable for display on a se- 
lected display device. 
Description of the Related Art 

[0002] Display devices generally include a display screen includ- 
ing a number of horizontal lines. The number of horizon- 
tal and vertical lines defines the resolution of the corre- 
sponding digital display device. Resolutions of typical 
screens available in the market place include 640 x 480, 
1024 x 768 etc. In order to display a source image on a 
display screen, each source image is transmitted as a se- 



quence of frames each of which includes a number of 
horizontal scan lines. Typically, a time reference signal, or 
signals, is provided in order to divide the analog signal 
into horizontal scan lines and frames. In the VGA/SVGA 
environments, for example, the reference signals include 
a VSYNC signal and an HSYNC signal where the VSYNC 
signal indicates the beginning of a frame and the HSYNC 
signal indicates the beginning of a next source scan line. 
Therefore, in order to display a source image, the source 
image is divided into a number of points and each point is 
displayed on a pixel in such a way that point can be rep- 
resented as a pixel data element. Display signals for each 
pixel on the display may be generated using the corre- 
sponding display data element. 
[0003] For example, Fig. 1 illustrates a conventional NTSC stan- 
dard TV displayed image 100. The image 100 is formed of 
an active picture 10 that is the area of the image 100 that 
carries image information. Outside of the active picture 10 
is a blanking region 11 suitable for line and field blanking. 
The active picture 10 uses frames 12, pixels 14 and scan 
lines 16 to form the actual TV image. The frame 12 repre- 
sents a still image produced from any of a variety of 
sources such as an analog video camera, an analog televi- 



sion, etc. 

[0004] information in frame 12 is represented by any number of 
pixels 14. A pixel ( an acronym for "picture element") is 
the smallest distinguishable and resolvable area in an im- 
age as well as the discrete location of an individual photo- 
sensor in a solid state camera. Each pixel in turn repre- 
sents digitized information and is often represented by 8 
bits, although each pixel may be represented by any num- 
ber of bits. Each scan line 16 includes any number of pix- 
els 14, thereby representing a horizontal line of informa- 
tion within frame 12. In NTSC video (a television standard 
using interlaced scan), for example, a field of information 
appears every 60th of a second, a frame (including 2 
fields) appears every 30th of a second and the continuous 
presentation of frames of information produce a picture. 
On a computer monitor using progressive scan, a frame of 
information is periodically refreshed on the screen to pro- 
duce the display seen by a user. 

[0005] The number of frames-per-second (fps) is also an essen- 
tial factor in the perception of a moving image. Films are 
shot at 24 Fps and usually displayed at movie theaters re- 
peating each frame two times for a net 48 fps to avoid 
flickering. NTSC television uses 60 interlaced fields (fps) 



per second and PAL uses 50 fps. The interlaced fields are 
displaced one vertical line and happen at two different in- 
stances in time, they are called even field and odd field al- 
ternatively. The 60 fps can be perceived as a single com- 

th 

plete frame every 30 of a second whereas film is 
scanned progressively as a complete frame. Most internet 
media today uses 15 fps and useable moving images can 
have a 10 Fps frame rate. 

[0006] | n order to display these various video formats in a single 
display, the various video streams must be processed into 
a single video stream having video format consistent with 
a display device, such as a monitor or TV, on which the 
images are to be displayed. This is particularly important 
when attempting to display images from such disparate 
sources as an NTSC TV source (which is continuous in na- 
ture) at 60 fps interlaced or 30 fps progressive and inter- 
net media (which is packet based) at 15 fps or even lower. 
Additionally, it would be advantageous to integrate the 
requisite video processing into the display itself in order 
to provide a cost effective solution. 

[0007] Therefore what is desired is an efficient method and ap- 
paratus for processing any of a number of multi-format 
data streams (including video data and network data) into 



a single format data stream suitable for display on a mon- 
itor 

Summary of Invention 

[0008] According to the present invention, methods, apparatus, 
and systems are disclosed for processing a number of 
multi-format video data streams into a single synchro- 
nized display video stream. 

[0009] a configurable real time data processor arranged to pro- 
vide a data stream to a display unit having an associated 
set of display attributes. A number of ports each of which 
is configured to receive an input data stream, a number of 
adaptive image converter units each of which are coupled 
to a corresponding one of the ports suitable for convert- 
ing a corresponding input data stream to a corresponding 
converted data stream having associated converted data 
stream attributes, an image compositor unit arranged to 
combine the converted data streams to form a composited 
data stream, an image enhancer unit arranged to enhance 
the composited data stream to form an enhanced data 
stream, and a display unit interface arranged process the 
enhanced data stream suitable for display on the display 
unit. 

[0010] a method of adaptively providing a data stream to a dis- 



play unit having an associated set of display attributes. 
Receiving a number of input data streams at a number of 
corresponding input ports, converting the input data 
streams to a corresponding converted data stream having 
associated converted data stream attributes, compositing 
the converted data streams by an image compositor, en- 
hancing the composited data stream, and processing the 
enhanced data stream for display on the display unit. 
[0011] Computer program product for adaptively providing a 

data stream to a display unit having an associated set of 
display attributes, computer code for receiving a number 
of input data streams at a number of corresponding input 
ports, computer code for converting the input data 
streams to a corresponding converted data stream having 
associated converted data stream attributes, computer 
code for compositing the converted data streams by an 
image compositor, computer code for enhancing the 
composited data stream, computer code for processing 
the enhanced data stream for display on the display unit, 
and computer readable medium for storing the computer 
code. 

Brief Description of Drawings 
[0012] The invention will be better understood by reference to 



the following description taken in conjunction with the 
accompanying drawings. 
[0013] pig. 1 illustrates a conventional NTSC standard TV picture. 

[0014] pig. 2 shows a representative embodiment of the inven- 
tion implemented as a video processing circuit having a 
multi-format video receiver port, a user interface port, 
and a network interface. 

[0015] Fig. 3 shows a flowchart detailing a process for concur- 
rently processing a number of video data streams in ac- 
cordance with an embodiment of the invention. 

[0016] Fig. 4 illustrates a computer system employed to imple- 
ment the invention. 
Detailed Description of Selected Embodiments 

[0017] Reference will now be made in detail to a particular em- 
bodiment of the invention an example of which is illus- 
trated in the accompanying drawings. While the invention 
will be described in conjunction with the particular em- 
bodiment, it will be understood that it is not intended to 
limit the invention to the described embodiment. To the 
contrary, it is intended to cover alternatives, modifica- 
tions, and equivalents as may be included within the spirit 
and scope of the invention as defined by the appended 



claims. 

[0018] | n one embodiment, an integrated video processor suit- 
able for concurrently processing any of a number of video 
data streams having an associated video format for dis- 
play on a selected video display unit at a selected video 
format is described. The video processor includes any of a 
number of input ports, that includes, but is not limited to, 
a multi-format video port, a user interface port, and a 
network interface. In the described embodiment, any of a 
number of multi-format video streams received by the 
multi-format video port are converted by way of a format 
converter unit to a progressive scan video format. Such 
formats include component, composite, serial digital, par- 
allel digital, RGB, or consumer digital video. The digital 
video signal can be any number and type of well known 
digital formats such as, SMPTE 274M-1995 (1920 x 1080 
resolution), SMPTE 296M-1997 (1280 x 720 resolution), 
as well as standard 480 progressive scan video. The out- 
puts of the video format converter unit, the user interface 
port and the network interface are each supplied to a cor- 
responding image converter unit that assures that each 
signal provided to an image compositor unit is the same 
format consistent with the display. It should be noted that 



in the case of interlaced input data, the format converter 
unit provides a de-interlacing function that converts an 
interlaced image to a non-interlaced image (i.e., progres- 
sive scan type image). In those situations, however, where 
an interlaced image is to be displayed, an interlacing unit 
described below is used to appropriately interlace the im- 
age. 

[0019] The image compositor unit, in turn, combines each of the 
provided signals to a single video data stream suitable for 
display on the display unit. In the described embodiment, 
the single video data stream is input to a video enhancer 
unit arranged to provide selected enhancement algorithms 
to the video data stream. Such enhancements include 
edge correction, contrast enhancement, etc. The en- 
hanced video signal is, in turn, provided to a display unit 
interface that includes a progressive bypass which by- 
passes an interlacer unit included therein in those cases 
where the display is configured to display a progressive 
scan type image. 

[0020] | n the described embodiment, the inventive processor is 
incorporated in a integrated circuit or other such device in 
such a way as to enable the processor to be incorporated 
within the display without requiring a separate unit. In this 



way, a video receiver so equipped can directly receive and 
display in any selected format video data from any num- 
ber and kind of video source such as satellite, cable, 
packetized network data, and the like. 

[0021] The invention will now be described in terms of a real time 
input video data stream processing unit suitable for inte- 
gration with a video display system. It should be noted, 
however, that the described embodiments are for illustra- 
tive purposes only and should not be construed as limit- 
ing either the scope or intent of the invention. 

[0022] Accordingly, Fig. 2 shows a representative embodiment of 
the invention implemented as a video processing circuit 
200 having a multi-format video receiver port 202, a user 
interface port 204, and a network interface port 206. In 
the described embodiment, the video processing circuit 
200 is incorporated directly into a display device 208 hav- 
ing a display 210 suitable for displaying any images pro- 
vided thereto in a particular video format. For example, in 
those cases where the display 210 is a CRT progressive 
scan type display, then only progressive scan type video 
signals can be displayed whereas in those cases where the 
display 210 is a conventional interlaced type display, then 
only interlaced type video signals are suitable to be dis- 



played. 

[0023] Therefore, in those cases where the video processing cir- 
cuit 200 is directly incorporated into the display device 
208 having a dedicated display unit, then the video pro- 
cessing unit 200 provides a video signals that are appro- 
priate only for the dedicated display and no other. How- 
ever, in those cases where the video processing circuit 
200 is not directly incorporated into the display device 
208 but is nonetheless capable of being coupled to the 
display device, the inventive circuit 200 can be used to 
process video signals for any of a number of different type 
displays each arranged to display video signals of a corre- 
sponding format. In these cases then the video processing 
circuit 200 is a configurable video processing circuit. In 
those cases where the video processing circuit 200 is con- 
figurable, the display unit 210 provides a set of display 
attributes 212 (such as color space, progressive vs inter- 
laced, resolution, refresh rate, etc.) to a system controller 
unit 214. It should be noted that the display attributes can 
be described in terms of Extended Display Identification 
Data (EDID) that is a VESA standard data format that con- 
tains basic information about a monitor and its capabili- 
ties, including vendor information, maximum image size, 



color characteristics, factory pre-set timings, frequency 
range limits, and character strings for the monitor name 
and serial number. The system controller unit 214 uses 
the set of display attributes 212 to configure the various 
elements of the video processing circuit 200 in order to 
provide a video signal of the appropriate kind and format 
for display by the display 210. For example, when the dis- 
play device 208 is a digital television, then the video sig- 
nal is a digital video signal having any number and type of 
well known digital formats such as, SMPTE 274M-1995 
(1920 x 1080 resolution, progressive or interlaced scan), 
SMPTE 296M-1997 (1280 x 720 resolution, progressive 
scan), as well as standard 480 progressive scan video and 
graphics. 

[0024] An image source 216 coupled to the multi-format video 
port 202 provides any number of digital or analog image 
input signals for processing by the circuit 200. The image 
source 216 can provide a digital image stream that can 
take the form of a still image (having a format such as 
JPEG or TIFF) as well as video from, for example, a DVD 
player, set top box (with satellite DSS or cable signal ) and 
the like. In this way, the image source 216 can provide 
any number and type of well-known digital formats, such 



as, JPEG, BMP, TIFF, BNC composite, serial digital, parallel 
digital, RGB, or consumer digital video. 

[0025] As well known in the art, a television signal generally in- 
cludes display data and corresponding synchronization 
signals. The display data usually represents color intensity 
for different points and the synchronization signals pro- 
vide a time reference such that each point is associated 
with a point of an image. Synchronization signals typically 
include horizontal synchronization signals separating 
each line and vertical synchronization signals separating 
each frame. Each frame usually corresponds to an image 
and frames are encoded at 60 Hz in conventional televi- 
sion signals according to NTSC format known in the art. 

[0026] | n many instances digital data is encoded in television sig- 
nals. For example, digital data is often encoded in the 
vertical blanking interval (VBI) of a television signal (VBI 
generally refers to the time duration or signal portion be- 
tween frames). The VBI duration provides sufficient time 
for the scan electronics of a (CRT based) television system 
to move a scan position to point from the bottom end of a 
display screen to the top. The television signal corre- 
sponding to the VBI period typically does not contain any 
display data (or image data), and thus a television signal 



portion corresponding to the VBI period has been conve- 
niently used to encode digital data. 

[0027] using the ability to encode digital data in a television sys- 
tem, broadcasters (or television signal generators, in gen- 
eral) may send data corresponding to several applications 
useful for viewers. For example, information is often en- 
coded in the VBI to enable the display of selected text on 
television displays. Some companies broadcast television 
guide (indicating the program schedule) and some other 
companies provide stock quotes and news flashes using 
VBI portion of a television signal. Digital data can be en- 
coded in television signal portions other than VBI also. For 
example, an entire channel of a television signal can be 
used to encode teletext data. Accordingly, the multi-for- 
mat video receiver 202 includes a number of sub-circuits 
arranged singly or in combination to perform a number of 
functions that include, for example, a video decoder cir- 
cuit, a digitization circuit, an MPEG decoder circuit, an RF 
decoder circuit and a VBI decoder circuit. 

[0028] The user interface port 204 provides access to the circuit 
200 for a user input device 218 that can take the form of 
a remote control device. As a remote control device, a 
user can invoke specific user supplied instructions (such 



as navigation control, volume, brightness, contrast, etc.) 
that are used, in turn, to control various aspects of the 
displayed image. In other situations, the user interface can 
enable closed captioning suitable for display of textual in- 
formation to be incorporated into the display. Addition- 
ally, many user input devices provide navigation control 
signals used for navigating various on-screen displays 
(OSD) such as menus for DVDs, channel guides, and the 
like. In this way, the data provided by the user input de- 
vice 218 is typically asynchronous in nature. 
[0029] The network interface 206 provides a bi-directional link 
between network applications and data provided by a net- 
work (such as the Internet, intranets, LANs, WANs, etc.) 
and the inventive circuit 200. In most cases, the data pro- 
vided by the network to the network interface 206 is 
packetized in nature along the lines of ATM data packets, 
Ethernet data packets, TCP/IP protocol type data packets 
and the like. In order, therefore, to integrate what is es- 
sentially discrete data packets consistent with that re- 
ceived by the network interface 206, the packetized data 
must be decompressed and depacketized by a depacke- 
tizer 220 included in or coupled to the network interface 
206 and a memory unit 231. 



[0030] ^ should be noted that each of the data streams from 

each of the ports has a clock associated with it. For exam- 
ple, in the case of the multi-format video receiver port 
202, a video clock 4> , in the case of the user interface, a 

vid 

user interface clock d> , and in the case of the network in- 

ui 

terface, a network clock d> (for example, input video can 

net 

interlaced and lower resolution (i.e. 720 x 480 I) whereas 
the network data could be progressive and higher resolu- 
tion (i.e., 1024 x 768 P). In particular, in the video clock d> 

vid 

can represent the frame rate of any incoming video signal 
(such as 30 frames per second (fps) progressive or 60 fps 
interlaced) whereas the network clock video clock d> can 

net 

be 15 fps. For example, cb could be 60 Hz, d> (such as 

vid net 

from a PC) could be 72 Hz, whereas d> could be 75 Hz. 

ui 

[0031] An input format converter unit 221 coupled to the output 
of the multi-format video receiver port 202 is configured 
to convert the incoming video data streams to a progres- 
sive scan type video format, if necessary. In those cases 
where the incoming data stream is already a progressive 
scan type format, the de-interlacing function is bypassed 
altogether. As part of the input format converter unit 221, 
a de-interlacer sub circuit provides for conversion of in- 
terlaced video data to progressive video data whereas in 



those cases where the input video data is already progres- 
sive video data, a progressive bypass circuit bypasses the 
interlacer. 

[0032] a number of image converter blocks 222 through 226 are 
provided to convert input progressive scan data streams 
to a progressive output image size and timing based upon 
a progressive clock provided <t> by a progressive dis- 

prog 

play clock 228 that runs at the display rate of the display 
210. In addition to converting the input video data 
streams, each of the image converter blocks interfaces 
with the memory unit 231 (or a memory controller unit if 
included therein). In this way, each of the image converter 
units can write the input video data directly to the mem- 
ory unit 231 or provide processed image data into the 
memory unit 231. In the described embodiment, the im- 
age converter blocks include sub circuits taken singly on 
in combination that function as a horizontal scaler, a ver- 
tical scaler as well as a temporal scaler. The temporal 
scaler is arranged to provide frame rate conversion using 
various sub circuits taken singly or in combination to per- 
form selected video processes such as any number and 
type of well known motion compensation techniques. For 
example, in the case of data provided by the user inter- 



face 218 that will be part of the displayed image, the as- 
sociated image converter block 224 must provide at least 
a temporal scaler in order to match the displayed user in- 
put information to the frame rate of the displayed image. 

[0033] | n some cases, it would be advantageous to process any 
video data prior to the data being stored in the memory 
unit 231 such as in those cases when the video images 
are being downscaled. Therefore, by downscaling the 
video data prior to storing in the memory unit 231, sub- 
stantial memory resources are conserved due to the re- 
duced number of pixels and/or data per pixel required 
post processing. Such a situation is picture in picture (PIP) 
where a larger image is downscaled to a small PIP window. 

[0034] | n addition to storing video data in the memory unit 231, 
each of the image converter blocks has the ability to read 
video data from the memory unit 231 and processes the 
data accordingly. In either case, each of the image con- 
verter blocks can then be used to pass the video data read 
from the memory unit 231 to an image compositor unit as 
needed. If the frame rates are different, then frame rate 
conversion is performed by writing video data into the 
memory at a first frame rate and read out at a display 
rate. By providing memory resources to each of the image 



converter units, the output frame rates have the same 
clock which could be locked to any of the input video data 
streams (i.e., d> ,d> , or dp ) or could be free running 

vid ui prog 

in that each data stream comes in at its own rate but is 
locked to a display rate or ration of rates that may be than 
any of the incoming rates. 
[0035] An image compositor 230 requests video data from se- 
lected ones of the image converter blocks 222- 226. In 
the described embodiment, the image compositor 230 in- 
tegrates all video signals provided thereto regardless of 
the source. Since all input video signals are now the same 
format and the same clock, the image compositor 230 
forms an output video stream by combining each of the 
input video signals based upon a control signal provided 
by the system controller unit. The requested video data is 
then composited in such as way to form an output video 
data stream 232 that, in turn, is provided to a video en- 
hancer unit 234 arranged to provide selective enhance- 
ment algorithms to the video data stream. Such enhance- 
ments include edge correction, contrast enhancement, 
sharpness control, color manipulation and control, bright- 
ness, either adaptively or under user control and de- 
scribed in more detail in issued U.S. patents 5940141, 



5844617, 5237414, 5151783, 5014119, 4939576, and 
4847681 each of which are incorporated by reference for 
all purposes. The enhanced video signal is, in turn, pro- 
vided to a display unit interface 236 that includes a pro- 
gressive bypass which bypasses an interlacer unit in- 
cluded therein in those cases where the display 210 is 
configured to display a progressive scan type image. 
[0036] The invention will now be described in terms of a process 
300 as illustrated by a flowchart shown in Fig. 3 that 
shows a flowchart detailing a process for concurrently 
processing a number of video data streams in accordance 
with an embodiment of the invention. Accordingly, the 
process 300 begins at 302 by a determination of whether 
or not a set of display attributes is to be updated. If it is 
determined that the set of display attributes is to be up- 
dated, then a set of display attributes are provided at 304 
and, based upon the display attributes, appropriate video 
processing elements are configured at 306. In any case, at 
308 input video data is received and at 310 the input 
video data is converted based upon the display attributes. 
At 312, an image compositor composites selected por- 
tions of the converted video data while at 314, the com- 
posited video data is selectively enhanced at 316. At 318, 



the enhanced video data is displayed on a display unit. 
[0037] pig. 4 illustrates a computer system 400 employed to im- 
plement the invention. Computer system 400 is only an 
example of a graphics system in which the present inven- 
tion can be implemented. Computer system 400 includes 
central processing unit (CPU) 810, random access memory 
(RAM) 420, read only memory (ROM) 425, one or more 
peripherals 430, graphics controller 460, primary storage 
devices 440 and 450, and digital display unit 470. As is 
well known in the art, ROM acts to transfer data and in- 
structions uni-directionally to the CPU 410, while RAM is 
used typically to transfer data and instructions in a bi- 
directional manner. CPU 410 may generally include any 
number of processors. Both primary storage devices 440 
and 450 may include any suitable computer-readable me- 
dia. A secondary storage medium 480, which is typically a 
mass memory device, is also coupled bi-directionally to 
CPU 410 and provides additional data storage capacity. 
The mass memory device 480 is a computer-readable 
medium that may be used to store programs including 
computer code, data, and the like. Typically, mass mem- 
ory device 480 is a storage medium such as a hard disk or 
a tape which generally slower than primary storage de- 



vices 440, 450. Mass memory storage device 480 may 
take the form of a magnetic or paper tape reader or some 
other well-known device. It will be appreciated that the 
information retained within the mass memory device 480, 
may, in appropriate cases, be incorporated in standard 
fashion as part of RAM 420 as virtual memory. 

[0038] Finally, CPU 410 optionally may be coupled to a computer 
or telecommunications network, e.g., an Internet network 
or an intranet network, using a network connection as 
shown generally at 495. With such a network connection, 
it is contemplated that the CPU 410 might receive infor- 
mation from the network, or might output information to 
the network in the course of performing the above- 
described method steps. Such information, which is often 
represented as a sequence of instructions to be executed 
using CPU 410, may be received from and outputted to 
the network, for example, in the form of a computer data 
signal embodied in a carrier wave. The above-described 
devices and materials will be familiar to those of skill in 
the computer hardware and software arts. 

[0039] Graphics controller 460 generates analog image data and 
a corresponding reference signal, and provides both to 
digital display unit 470. The analog image data can be 



generated, for example, based on pixel data received from 
CPU 410 or from an external encode (not shown). In one 
embodiment, the analog image data is provided in RGB 
format and the reference signal includes the VSYNC and 
HSYNC signals well known in the art. However, it should 
be understood that the present invention can be imple- 
mented with analog image, data and/or reference signals 
in other formats. For example, analog image data can in- 
clude video signal data also with a corresponding time 
reference signal. 

[0040] Although only a few embodiments of the present inven- 
tion have been described, it should be understood that 
the present invention may be embodied in many other 
specific forms without departing from the spirit or the 
scope of the present invention. The present examples are 
to be considered as illustrative and not restrictive, and the 
invention is not to be limited to the details given herein, 
but may be modified within the scope of the appended 
claims along with their full scope of equivalents. 

[0041] while this invention has been described in terms of a pre- 
ferred embodiment, there are alterations, permutations, 
and equivalents that fall within the scope of this invention. 
It should also be noted that there are may alternative ways 



of implementing both the process and apparatus of the 
present invention. It is therefore intended that the inven- 
tion be interpreted as including all such alterations, per- 
mutations, and equivalents as fall within the true spirit 
and scope of the present invention. 



